Feature selection - one form of dimension reduction

Even if \(X\) full rank, we might want eliminate some predictors to decrease model variance.

• near - zero variance predictors

• highly correlated predictors (with each other)

• predictors uncorrelated with the response

The goal of dimension reduction

Curse of dimensionality

Curse of dimensionality

Suppose KNN with \(p\) predictors \(\sim U(0,1)\)

• volume of hypercube with sides of length \(d\) is \(d^p\).

• To capture \(r\) proportion of the data, we want \(d^p = r\), so \(d = r^{1/p}\)

Principal components and eigenvectors

First scale or standardize the data matrix.

The principal components - the columns of \(\mathbf{U}\) - are the eigenvectors of \(\mathbf{X}^T \mathbf{X}\).

(Equivalently, the eigenvectors of the correlation matrix - standardizing is automatic.)

Theorem

• Eigenvectors of \(\mathbf{X}^T \mathbf{X}\) are orthogonal

• Let \[ \lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_p > 0 \] with eigenvectors \(\mathbf{v}_1, \dots, \mathbf{v}_p\)

  • \(\lambda_i\) is variance in direction \(\mathbf{v}_i\)
  • \(\mathbf{v}_1\) maximizes variance
  • \(\mathbf{v}_2\) maximizes variance subject to orthogonality with \(\mathbf{v}_1\)
  • and so on

Principal componponents regression (PCR)

1. Center and scale the data

2. Form \(\mathbf{U}\), the eigenvectors of centered and scaled \(\mathbf{X}^T \mathbf{X}\), ordered by decreasing eigenvalues

3. Compute \(\mathbf{Z} = \mathbf{XU}\)

4. Select the number of components, \(m\) based on explained variance (or cross-validation).

5. Fit a linear regression using the first \(m\) columns of \(\mathbf{Z}\) as predictors.

Example: wine data (UCI ML Repository)

Chemical analysis of wines grown in the same region in Italy but derived from three different cultivars. The analysis determined the quantities of 13 constituents found in each of the three types of wines.

Multinomial regression on the PCA features

We get perfect score with just 5 components, and 98% with only 3 components.

PCA as a preprocessing step

• on select predictors - not all or nothing.

• preprocessing/feature engineering step for any ML model

• Use tidymodels::recipes, sklearn PCA, prcomp

PCA on USArrests dataset¹

Loadings and scores

[1] 0.620060 0.867502 0.956642 1.000000

Visualizing the loadings

Biplots

We can visualize the loadings and the scores together with a biplot.

Other matrix factorizations for feature extraction

• Non-negative Matrix Factorization (NNMF)

• Independent Component Analysis (ICA)

Uniform manifold approximation and projection (UMAP)

• McInnes, L., Healy, J., & Melville, J. (2018)

• UMAP python package: umap-learn

• R package: umap, can use tidymodels recipes step.

Constructs a graph representation of the data in high dimensions, then optimizes a low-dimensional graph to be as structurally similar as possible.

Uniform manifold approximation and projection (UMAP)

penguins revisited

PCA vs UMAP on penguins

UMAP may be overkill here

UMAP

mnist dataset - images of handwritten digits (0-9). 784 features (28 x 28 pixels)

PCA vs UMAP on mnist

mnist dataset - images of handwritten digits (0-9). 784 features (28 x 28 pixels)

More on UMAP

• umap-learn docs

• A Coenen, A. Pearce blog Understanding UMAP

Comparing PCA, UMAP, and t-SNE

Source: Biostatsquid

Neural Networks

Source: Gerhard Paab blog

Source: S. Raschka Deep Learning Learning

Coding notes

• prcomp in base R, step_pca() recipes step in recipes package (part of tidymodels)

• sklearn.decomposition.PCA

• umap package in R; as a recipes() step

• umap-learn not part of sklearn but part of sklearn ecosystem

See my slides source code as well as example R and ipynb files for example code.

Recap

• Why dimension reduction matters

  • High-dimensional predictors and rank deficiency
  • Variance, multicollinearity, interpretability, computational cost
  • The curse of dimensionality (analytical + simulation intuition)

• Feature selection, engineering, extraction

• Principal Components Analysis (PCA)

  • Mathematics; eigenvectors and eigenvalues
  • Principal components regression (PCR)
  • PCA loadings and scores

• Nonlinear dimension reduction

  • UMAP, neural networks